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Abstract 

Objective. Stakeholders of hospitals often lack standardized tools to assess compliance with quality management strategies and 
the implementation of clinical quality activities in hospitals. Such assessment tools, if easy to use, could be helpful to hospitals, 
health-care purchasers and health-care inspectorates. The aim of our study was to determine the psychometric properties of two 
newly developed tools for measuring compliance with process-oriented quality management strategies and the extent of imple- 
mentation of clinical quality strategies at the hospital level. 

Design. We developed and tested two measurement instruments that could be used during on-site visits by trained external sur- 
veyors to calculate a Quality Management Compliance Index (QMCI) and a Clinical Quality Implementation Index (CQII). We 
used psychometric methods and the cross-sectional data to explore the factor structure, reliability and validity of each of these 
instruments. 

Setting and Participants. The sample consisted of 74 acute care hospitals selected at random from each of 7 European 
countries. 

Main Outcome Measures. The psychometric properties of the two indices (QMCI and CQII). 

Results. Overall, the indices demonstrated favourable psychometric performance based on factor analysis, item correlations, in- 
ternal consistency and hypothesis testing. Cronbach's alpha was acceptable for the scales of the QMCI (a: 0.74—0.78) and the 
CQII (a: 0.82—0.93). Inter-scale correlations revealed that the scales were positively correlated, but distinct. All scales added suf- 
ficient new information to each main index to be retained. 

Conclusion. This study has produced two reliable instruments that can be used during on-site visits to assess compliance with 
quality' management strategies and implementation of quality management activities by hospitals in Europe and perhaps other 
jurisdictions. 

Keywords: quality management, audit, implementation, on-site visits, hospital 



Introduction 

In a more and more market-oriented health-care delivery system 
it may be increasingly important to evaluate the quality of care 
delivered. To this end, health-care purchasers are gathering and 
using performance information on patient experiences as well 



as organizational and clinical performance indicators. Despite 
years of efforts to improve the reliability and validity of per- 
formance indicators, there are still differences in outcomes 
related to registration and measurement error in the administra- 
tive databases used to calculate outcome indicators. Additional 
information about quality management and clinical quality 7 
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strategies could be a valuable adjunct to hospital assessment, es- 
pecially since accreditation/ certification instruments that can be 
used to assess quality management systems already exist [1]. 

An alternative to administrative data and surveys is the 
on-site visit (or audit). Visits involving independent auditors 
can verify compliance with activities, methods and procedures 
used to plan, control, monitor and improve the quality of care. 
On-site visits are mainly used by accreditation and certification 
organizations who visit an organization for a few days, review- 
ing documents and meeting with front-line staff. To date, 
easy-to-use survey instruments have not been developed for 
use by health-care purchasers or health-care inspectorates. 
Such instruments could be used to reveal whether a hospital 
has appropriate quality management strategies in place, 
whether they are used and whether they are stimulating 
continuous learning and improvement. The latter is based on 
the Deming or Nolan Quality Improvement Cycle, which 
describes the steps: Plan— Do— Check/ study— Act. On-site 
visits offer an opportunity to discuss achievement of more 
complex steps like 'Check and Act'. In addition, widely used 
quality indicators can rarely be used to measure the improve- 
ment structures and culture of hospital units, which can be 
explored in conversation during an on- site-visit. 

In this article we describe the development of two novel 
quality management indices for purchasers and other stake- 
holders of European hospitals. Both are developed within the 
DUQuE project (Deepening our understanding of Quality 
Improvement in Europe). One (the Quality Management 
Compliance Index or QMCI) focuses on compliance with 
existing quality management procedures at the hospital level 
and the other (the Clinical Quality Implementation Index or 
CQII) on activities that support continuous improvement of 
clinical indicators. This paper describes testing of the psycho- 
metric properties of the two newly developed measurement 
instruments (QMCI and CQII). 

Methods 

Setting and participants 

The study took place in the context of the DUQuE project 
which ran from 2009 to 2013 [2]. Hospitals were sampled at 
random from all acute care hospitals in seven European coun- 
tries: Czech Republic, France, Germany, Poland, Portugal, 
Spain and Turkey. Data for the QMCI and CQII were col- 
lected in the hospitals that participated in the in-depth study of 
the DUQuE project. In total, 74 hospitals (response rate 88%) 
were visited by experienced surveyors and these responses 
were used for the psychometric analyses. 

Development of the instruments 

Quality Management Compliance Index. The aim of the QMCI 
was to identify and verify the compliance to a set of closely 
related methods and procedures used to plan, monitor and 
improve the quality of care. By design, three scales of the index 
were defined a priori as quality planning, monitoring opinions 



of professionals and patients, and improvement of quality of 
care based on the notion that if a hospital would like to base 
its quality management on a limited number of activities, then 
it would need to have a plan based on the opinions of front-line 
staff and the users of hospital services, patients. Furthermore, 
strategies are necessary to solve possible shortcomings mentioned 
by professionals and patients. 

The choice of questionnaire items was based on expert 
opinion of experts with years of experience in hospital per- 
formance evaluation during accreditation and certification 
audits. The main criteria for including an item were its 
assumed influence on quality and safety of care, and the feasi- 
bility of verifying an answer to the item. Face validity was 
established based on the review by 10 experts of the DUQuE 
project, and a pilot test in two hospitals. All items of the 
QMCI (« =15) were rated on a five-point Likert scale, varying 
from 'no or negligible compliance' (0) to 'full compliance' (4). 

Clinical Quality Implementation Index. The purpose of the 
CQII is to test clinical quality systems and seek evidence of their 
implementation at the hospital level. The CQII has been 
designed to measure to what extent efforts regarding key clinical 
quality areas are implemented across the hospital. Following 
Bate and Mendel [3], each quality effort is assessed with regard 
to three levels of development: (i) Do quality efforts regarding 
the key areas exist (i.e. is there a responsible group and hospital 
protocol)? (ii) To what extent are these efforts monitored (i.e. 
with regard to compliance and improvement measurements)? (iii) 
To what extent is the sustainability of these efforts monitored? 

The key clinical areas included stem from the different 
quality functions described in most accreditation systems, as 
well as the recommendations of the WHO Patient Safety 
Alliance covering most of the key hospital clinical and safety 
areas. In total, seven areas were selected: (i) preventing hospital 
infection, (ii) medication management, (iii) preventing patient 
falls, (iv) preventing pressure ulcers, (v) routine assessment 
and diagnostic testing of patients in elective surgery, (vi) safe 
surgery that includes an approved checklist and (vii) preventing 



Table I Characteristics of hospitals participating in the 
analysis 



Characteristic 


n 


% 


Number of hospitals 


74 


100 


Teaching status, n (%) 






Teaching 


33 


45 


Non-teaching 


41 


55 


Ownership, n (%) 






Public hospitals 


59 


80 


Private (or mixed ownership) 


15 


20 


Approximate number of beds in hospital 






<200 


7 


9 


200-500 


22 


30 


501-1000 


31 


42 


>1000 


14 


19 



28 



Table 2 Factor loadings, Cronbach's alpha and corrected item-total correlations of QMCI (» — 74) 



Scale and items of QMCI Factor loadings Internal consistency Corrected 

on primary scale reliability: Cronbach's a item-total 

correlation 



Quality planning 

Ql The hospital (management) board approved an annual programme for quality improvement in 2010 — 
Monitoring of patient/ professional opinions 

Q2 The results of patient satisfaction surveys were formally reported to the hospital (management) board 0.534 
in 2010 

Q3 The hospital (management) board received results of surveys of staff satisfaction in 2010 0.522 
Q4 Patients incidents and adverse events are analysed and evaluated 0.585 
Q5 Patients' opinion/perception is measured and evaluated 0.553 
Q6 Patient complaint system is available and/ or evaluated 0.534 
Q7 Professional opinion/perception is measured and evaluated 0.692 

Monitoring of quality systems 

Q8 The hospital (management) board received regular, formal reports on quality and safety in 2010 0.720 
Q9 Medical leaders received regular, formal reports on quality and safety in 2010 0.763 
Q10 There is an active clinical guidelines register 0.671 
Ql 1 Guidelines application are measured and evaluated 0.577 

Improving quality by staff development 

Q12 The hospital maintains a record for each member of the medical staff that contains a copy of documents 0.876 
related to license, education, experience and certification 

Q13 The hospital maintains a record for each member of the nursing staff that contains a copy of documents 0.847 
related to license, education, experience and certification 

Q14 The performance of all individual medical staff members is formally reviewed to determine continued 0.553 
competence to provide patient care services 

Q15 The performance of all nursing staff members is formally reviewed to determine continued competence 0.402 
to provide patient care services 



0.742 



0.783 



0.756 



0.450 

0.411 
0.491 
0.474 
0.440 
0.606 

0.593 
0.651 
0.607 
0.505 

0.674 

0.623 

0.545 

0.387 
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Table 3 Inter-scale correlation coefficients of QMCI and scales with overall construct (« = 74) 



1. 2. 3. 4. QMCI 

1 . Quality planning 1 0.78 

2. Quality control and monitoring of patient/professional opinions 0.317 1 0.69 

3. Quality control and monitoring of quality systems 0.520 0.475 1 0.78 

4. Improving quality by staff development 0.142 0.303 0.145 1 0.52 

Note. The numbers in the first row correspond to the scales in the first column. 
Table 4 Distribution of item and scale scores of QMCI (« = 74) 



Scale and items of QMCI (range 0—4) Median Floor (% with Ceiling (% with 

(IQR) a lowest score) highest score) 



Quality Management Compliance Index (QMCI) b (range 0—16) 


10 (3.2) 






Quality planning; mean (SD) 


2.9 (1.4) 






A 1-1-11 1 • 1 / ,\ 1 1 1 1 

Ql 1 he hospital (management) board approved an annual 


4(2) 


14 


58 


programme for quality improvement in 2010 








Monitoring of patient/professional opinions; mean (SD) 


2.7 (0.8) 






Qz 1 he results ot patient satisfaction surveys were formally reported 


4(2) 


7 


57 


to the hospital (management) board in 2010 








Q3 The hospital (management) board received results of surveys of 


2 (4) 


34 


o o 
28 


staff satisfaction in 2010 








Q4 Patients incidents and adverse events are analyzed and evaluated 


3 (3) 


19 


34 


Q5 Patients' opinion/perception is measured and evaluated 


4(1) 


1 


65 


Q6 Patient complaint system is available and/ or evaluated 


4(1) 


1 


69 


Q7 Professional opinion/perception is measured and evaluated 


2(4) 


35 


34 


Monitoring of quality systems; mean (SD) 


2.1 (1.1) 






Q8 The hospital (management) board received regular, formal reports 


3(2) 


14 


39 


on quality and safety in 2010 








Q9 Medical leaders received regular, formal reports on quality and 


3(2) 


15 


39 


safety in 2010 








Q10 There is an active clinical guidelines register 


2(4) 


27 


27 


Ql 1 Guidelines application are measured and evaluated 


1(2) 


32 


18 


Improving quality by staff development; mean (SD) 


2.4 (1.0) 






Q12 The hospital maintains a record for each member of the medical 


4(2) 


3 


61 


staff that contains a copy of documents related to license, education, 








experience and certification 








Q13 The hospital maintains a record for each member of the nursing 


4(2) 


3 


61 


staff that contains a copy of documents related to license, education, 








experience and certification 








Q14 The performance of all individual medical staff members is 


0(2) 


61 


18 


formally reviewed to determine continued competence to provide 








patient care services 








Q15 The performance of all nursing staff members is formally 


2(4) 


34 


38 



reviewed to determine continued competence to provide patient care 
services 



"Median (IQR) presented for individual question items. 
b QMCI is the sum of all 4 sub-scales, range: 0—16. 



deterioration and advance life support (i.e. rapid response 
teams, resuscitation programmes). As a whole, the audit in- 
strument comprised items from seven clinical areas, which 



could be rated on a five-point Likert scale, varying for example 
from 'no compliance' to 'full compliance'. Additionally, when 
no information was available 'not applicable' could be selected. 
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However, to get more meaning groups, the answer categories 
were recoded to a scale of 1—3, where responses of no, negli- 
gible or low compliance were coded as 1, medium compliance 
was coded as 2 and high, extensive or full compliance were 
coded as 3. The choice for the seven clinical areas was based 
on the fact that evidence exists on how to prevent the unsafe 
practices and related adverse outcomes. By following the existing 



Table 5 Factor loadings, Cronbach's alpha and corrected item- total correlations of CQII (« — 74) 



Scale and items of CQII 


Factor loadings 
on primary scale 


Internal consistency 
reliability: Cronbach's 
a 


Corrected 

item-total correlation 


Preventing hospital infection 




0.817 




CI Responsible group exists 


0.574 




0.522 


C2 Hospital protocol exists 


0.548 




0.491 


C3 Extent of compliance monitoring 


0.833 




0.789 


C4 Sustainability of the system 


0.808 




0.693 


C5 Improvement focus 


0.719 




0.558 


Medication management 




0.903 




C6 Responsible group exists 


0.671 




0.655 


C7 Hospital protocol exists 


0.567 




0.554 


C8 Extent of compliance monitoring 


0.954 




0.899 


C9 Sustainability of the system 


0.938 




0.855 


CIO Improvement focus 


0.917 




0.845 


Preventing patient falls 




0.898 




Cll Responsible group exists 


0.610 




0.590 


C12 Hospital protocol exists 


0.681 




0.648 


CI 3 Extent of compliance monitoring 


0.907 




0.859 


CI 4 Sustainability of the system 


0.952 




0.890 


CI 5 Improvement focus 


0.850 




0.772 


Preventing patient ulcers 




0.879 




CI 6 Responsible group exists 


0.644 




0.612 


CI 7 Hospital protocol exists 


0.631 




0.600 


CI 8 Extent of compliance monitoring 


0.867 




0.810 


CI 9 Sustainability of the system 


0.889 




0.807 


C20 Improvement focus 


0.804 




0.733 


Routine testing of elective surgery patients 




0.923 




C21 Responsible group exists 


0.581 




0.571 


C22 Hospital protocol exists 


0.702 




0.679 


C23 Extent of compliance monitoring 


0.984 




0.937 


C24 Sustainability of the system 


0.983 




0.929 


C25 Improvement focus 


0.970 




0.910 


Safe surgery practices 




0.881 




C26 Responsible group exists 


0.647 




0.616 


C27 Hospital protocol exists 


0.537 




0.513 


C28 Extent of compliance monitoring 


0.918 




0.850 


C29 Sustainability of the system 


0.887 




0.812 


C30 Improvement focus 


0.867 




0.809 


Preventing deterioration 




0.932 




C31 Responsible group exists 


0.804 




0.774 


C32 Hospital protocol exists 


0.787 




0.757 


C33 Extent of compliance monitoring 


0.905 




0.868 


C34 Sustainability of the system 


0.891 




0.850 


C35 Improvement focus 


0.895 




0.855 



guidelines patient harm might be prevented and patient safety 
improved. 

Data collection 

The QMCI and the CQII were designed as data collection 
tools for use during on-site visits by experienced surveyors 
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who had previous experience in hospital accreditation, but had 
no relationship with the hospitals. Data were collected in the 
74 hospitals between May 2011 and February 2012. In total, 
14 external surveyors (2 in each country) collected data. The 
surveyors were trained on the main aspects to be assessed, and 
the scoring system. A data collection manual was developed to 
provide guidance and ensure homogeneity of data collection. 
Data were first gathered on paper and then entered into an 
online database system and checked by the country coordin- 
ator for missing data. Every hospital was visited by two sur- 
veyors for 1 day. No hospital professionals were made aware 
of the contents of the visit beforehand. 



Data analysis 

We began the analysis by describing the sample of hospitals 
that provided data from external visits. Then, we investigated 
the factor structure, reliability and construct validity of QMCI 
and CQII using standard psychometric methods. We con- 
ducted principal components, confirmatory factor, reliability 
coefficient, item-scale total correlation and inter-scale correlation 
analyses separately for QMCI and CQII. There were no missing 
values for any of the items, as data collected from hospitals par- 
ticipating in external visits were complete. Since we had external 
visit data from only 74 hospitals and factor analysis required 5— 
10 observations per variable, we did not split the data into two 
parts to perform factor analysis. We explored factor structure 
using principal component analysis with oblique (promax) rota- 
tion with a factor extraction criterion of eigenvalues >1 and 
three or more item loadings. Items were assigned to the factor 
where they had the highest factor loading, and only items with 
loadings >0.3 were retained. However, only one item was used 
to assess the 'quality planning' domain for QMCI, and this was 
considered to be theoretically important for assessing quality 
compliance. We then used confirmatory factor analysis to 
examine whether the data supported the final factor structure, 
where root mean square residual <0.05 and a non-normed fit 
index >0.9 indicated good fit. We also used Cronbach's alpha to 
assess internal consistency reliability of each factor, where a value 
of 0.7 was acceptable. Item-total correlations corrected for item 
overlap were used to examine the homogeneity of each scale. 
Item-total correlation coefficients of 0.4 or greater suggested 
adequate scale homogeneity. Lastly, we assessed the degree of 



redundancy between scales using inter-scale correlation coeffi- 
cients, where a Pearson's correlation coefficient <0.7 was indica- 
tive of non-redundancy. Once psychometric evaluations of 
QMCI and CQII were completed and a final factor structure 
was established, we computed scores for each of the scales com- 
prising these indices by taking the mean of items retained for 
each scale from the factor analysis. These sub-scales were then 
summed to build each final index. We subtracted the number of 
scales from the final CQII in order to bring the lower bound of 
the index down to 0. In order to assess construct validity, we 
used Pearson's correlation coefficients to examine the relation- 
ship between CQII and QMCI. We also provide descriptive 
statistics on the final index, sub-scales aggregated to build the 
index and items that comprise the sub-scales. All statistical 
analyses were carried out in SAS (version 9.3, SAS Institute, Inc., 
NC, 2012). 

Results 

Hospital characteristics 

Across the 7 countries, 74 hospitals participated in this in-depth 
part of the DUQuE project. The teaching status was evenly 
balanced between non-teaching (55%) and teaching (45%). 
Most hospitals were publicly owned (80%) and comprised 501— 
1000 beds (42%) (Table 1). 

Quality Management Compliance Index 

The QMCI was designed to measure compliance in 3 
domains: quality planning (1 item); quality control and moni- 
toring (12 items) and improving quality by Staff development 
(5 items), but factor analysis revealed 4 factors instead of the 
proposed three, as can be seen in Table 2. The quality planning 
factor comprised one item as this domain was only assessed 
with one question in our questionnaire. The other three 
QMCI sub-scales were monitoring of patient/professional 
opinions, monitoring of quality systems and improving quality 
by staff development. Factor analysis revealed that the items 
initially included in the quality control and monitoring domain 
actually clustered on two distinct factors: the monitoring of 
the opinions of patients and professional and that of quality 
systems. This distinction is meaningful and was retained. 



Table 6 Inter-scale correlation coefficients (« — 74) 





1 


2 


3 


4 


5 


6 


7 


1 . Preventing hospital infection 


1.000 














2. Medication management 


0.585 


1.000 












3. Preventing patient falls 


0.130 


-0.019 


1.000 










4. Preventing patient ulcers 


0.249 


0.285 


0.391 


1.000 








5. Routine testing of elective surgery patients 


0.170 


0.204 


0.168 


0.371 


1.000 






6. Safe surgery practices 


0.364 


0.426 


0.227 


0.182 


0.276 


1.000 




7. Preventing deterioration 


0.371 


0.258 


0.285 


0.053 


0.160 


0.452 


1 


Overall construct CQII 


0.59 


0.60 


0.55 


0.60 


0.57 


0.70 


0.63 
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Table 7 Distribution of item, scale and index scores in — 74) 



Scale and items (range 1—3) 


Median (TQR) a 


Floor (% with 


Ceiling (% with 






lowest score) 


highest score) 


Clinical Quality Implementation Index (CQII) b (range 0-14) 


8.3 (2.9) 






Preventing hospital infection; mean (SD) 


2.8 (0.3) 






CI Responsible group exists 


3 (0.0) 


5 


88 


C2 Hospital protocol exists 


3 (0.0) 


1 


80 


C3 Extent of compliance monitoring 


3 (0.0) 


1 


93 


C4 Sustainability of the system 


3 (0.0) 


3 


89 


C5 Improvement focus 


3 (0.0) 


5 


86 


Medication management; mean (SD) 


2.4 (0.6) 






C6 Responsible group exists 


3 (1.0) 


16 


72 


C7 Hospital protocol exists 


3 (1.0) 


14 


61 


C8 Extent of compliance monitoring 


3 (1.0) 


20 


61 


C9 Sustainability of the system 


3 (1.0) 


20 


64 


CIO Improvement focus 


3 (1.0) 


23 


55 


Preventing patient falls; mean (SD) 


2.1 (0.7) 






Cll Responsible group exists 


2 (2.0) 


50 


39 


C12 Hospital protocol exists 


3 (1.0) 


23 


57 


CI 3 Extent of compliance monitoring 


3 (2.0) 


30 


55 


C14 Sustainability of the system 


2 (2.0) 


38 


49 


CI 5 Improvement focus 


2 (2.0) 


46 


45 


Preventing patient ulcers; mean (SD) 


2.3 (0.7) 






CI 6 Responsible group exists 


3 (2.0) 


30 


59 


CI 7 Hospital protocol exists 


3 (1.0) 


16 


58 


CI 8 Extent of compliance monitoring 


3 (2.0) 


26 


62 


CI 9 Sustainability of the system 


3 (2.0) 


28 


59 


C20 Improvement focus 


3 (2.0) 


35 


57 


Routine testing of elective surgery patients; mean (SD) 


1.5 (0.7) 






C21 Responsible group exists 


1 (1.0) 


66 


24 


C22 Hospital protocol exists 


1 (1.0) 


59 


22 


C23 Extent of compliance monitoring 


1 (1.0) 


72 


20 


C24 Sustainability of the system 


1 (1.0) 


72 


19 


C25 Improvement focus 


1 (1.0) 


74 


20 


Safe surgery practices; mean (SD) 


2.1 (0.7) 






C26 Responsible group exists 


3 (2.0) 


43 


50 


C27 Hospital protocol exists 


3 (1.0) 


20 


59 


C28 Extent of compliance monitoring 


2 (2.0) 


42 


46 


C29 Sustainability of the system 


2 (2.0) 


45 


45 


C30 Improvement focus 




J 1 


JJ 


Preventing deterioration; mean (SD) 


2.0 (0.8) 






C31 Responsible group exists 


3 (2.0) 


36 


55 


C32 Hospital protocol exists 


2 (2.0) 


32 


45 


C33 Extent of compliance monitoring 


2 (2.0) 


46 


43 


C34 Sustainability of the system 


2 (2.0) 


49 


43 


C35 Improvement focus 


2 (2.0) 


50 


36 



"Median (IQR) presented for individual question items 
b CQII is the sum of all 7 sub-scales (minus 7), range: 0—14. 



Three items from the questionnaire did not load on any of the 
factors and these were excluded. 

The factors of the QMCI yielded acceptable results with 
regard to internal consistency (Cronbach's alpha ranges 
between 0.74 and 0.78). None of the corrected item-total 



correlations were <0.4, except in the case of one item in the 
improving quality by staff development sub-scale, indicating 
that all items contribute to the distinction between high and 
low scores on the factor. The inter-scale correlations, 
presented in Table 3, had a maximum of 0.52, which is below 
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the maximum threshold of 0.70. This indicates that the QMCI 
is indeed a multi-dimensional construct with sub-scales 
addressing independent aspects of quality management. All 
sub-scales had notable correlations with the overall index, 
meaning that they contribute to the QMCI. 

Descriptive statistics for QMCI, the sub-scales and items 
that comprise the sub-scales are presented in Table 4. QMCI 
had a final scale range of 0—16. Six out of 15 items had a 
median score of 4 (range 0—4). This is also reflected in a high 
ceiling ratio of these items. In contrast to most other items, 
the third item (improving quality by staff development) was 
an exception. It had a low average (zero) and a high floor 
ratio. 

Clinical quality implementation index 

The CQII aimed to assess three levels of implementation, 
such as existence of protocol, monitoring of compliance and 
sustainability by measuring and using indicators to keep an im- 
provement focus. Factor analysis revealed, however, that the 
items did not group into these dimensions. Instead, the factors 
appear to be grouped according to different clinical areas 
(Table 5) suggesting that the levels of clinical implementation 
are not consistent on the same level across different clinical 
areas. Rather, the levels of development coexist and reflect the 
implementation of a certain area. Therefore, we used the items 
to describe clinical implementation as a single score for each 
area. The seven sub-scales retained by factor analysis were pre- 
venting hospital infection, medication management, prevent- 
ing patient falls, preventing patient ulcers, routine testing of 
elective surgery patients, safe surgery practices and preventing 
deterioration. The resulting seven-factor structure showed 
high factor loadings, Cronbach's alphas ranging from 0.82 to 
0.93 and corrected item-total correlations. The inter-scale cor- 
relations, presented in Table 6, had a maximum of 0.59, which 
is below the maximum threshold of 0.70. This indicates that 
the CQII is a multi-dimensional construct. 

CQII had a final scale range of 0—14. The distribution of 
the scores (Table 7) showed that the prevention of hospital infec- 
tion stands out with a very high average score and a ceiling ratio 
of over 80% for all of its items. For other items it seems that 
quite a number of hospitals have the highest or lowest score 
(ceiling effects). Around two-thirds of the hospitals have a low 
score on the sub-scale routine testing of elective surgery patients. 

Construct validity: hypothesis testing 

The inter-index correlation between QMCI and the CQII was 
0.565. This was in line with our expectations that both are dis- 
tinct, but related constructs. 



Discussion 
Main findings 

The results suggest that at the hospital level the QMCI and 
CQII are reliable and valid to assess compliance with quality 



management procedures as well as the extent of several activ- 
ities related to continuous improvement of clinical quality. The 
latter activities included having a group of professionals re- 
sponsible for the clinical area as well as a formally approved 
protocol and performance indicators. The initially proposed 
factor structure of both indices had to be adjusted based on 
the results of the factor analysis, but these minor adjustments 
did not change the theoretical constructs, instead refining the 
fit of the sub-scales to the concepts of interest. Both QMCI 
and CQII showed a high internal consistency and appeared 
to be multi-dimensional constructs. The descriptive results 
showed that some items included in the indices may be subject 
to ceiling effects with a large proportion of respondents having 
a positive score. Despite that, we kept the items and clinical 
areas in the instrument in case subsequent testing in across a 
broader range of hospitals in other European countries reveals 
more variation than those investigated in our study. 

Strength and limitations 

On-site audits have the advantage that they provide more ob- 
jective and independent outcomes based on factual informa- 
tion derived for instance from an annual report. In contrast to 
self-administered questionnaires, audit can avoid potential 
social desirability bias in responses. The time burden for the 
audited organization is relatively low compared with an all staff 
survey. The downside is that documents on which the audit is 
based have to be reliable. Furthermore, trained surveyors are 
needed to conduct the audit to minimize variation due to 
inter-observer differences. 

Relation with other studies 

Using audit as a measurement strategy is relatively rare, al- 
though it may grow as it is proving useful in other study 
designs [4, 5]. More often audit instruments are used as a tool 
to improve quality, especially in the form of an accreditation 
programme. As a recent review suggests, audits seem to lead 
to improved structure and process of care and even clinical 
outcomes [6]. 

Conclusion 

The two indices we developed and evaluated have the poten- 
tial for use in research and in routine practice to help hospi- 
tals focus on quality and safety issues as well as follow the 
quality improvement P-D-C/S-A cycle. The instruments can 
also be used by purchasers, policy-makers or health-care 
inspectorates, if they want to assess the implementation of 
quality management at hospitals level in a more standardized 
way. The QMCI focuses on the core elements of a quality 
system, while the CQII has its focus on clinical areas that are 
directly related to patient care at ward level. Future research 
is needed to investigate the relationship between these novel 
quality measurement tools and other indicators including 
patient outcomes. 
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